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Abstract 

Safety professionals typically do not engage in audits and independent assessments with the vigor as do our 
quality brethren. Taking advantage of industry and government experience conducting value added 
Independent Assessments or Audits benefits a safety program. Most other organizations simply call this 
process “internal audits.” Sources of audit training are presented and compared. A relation of logic 
between audit techniques and mishap investigation is discussed. An example of an audit process is offered. 
Shortcomings and pitfalls of auditing are covered. 

Introduction 


Audit definition - A documented systematic, independent, official, examination and verification of: records 
and other objective evidence of work performed; the process; or the process requirements to determine 
compliance to requirements; and to assess the effectiveness of implementation and identify potential 
improvements. This definition however may fall short of meeting objectives of the internal audit as 
discussed later in this paper. 

Auditing is one method for evaluating a design or process to ensure safety requirements are met. Auditing 
also offers a tried and proven means to improve the safety process. However, as with any tool or process, 
auditing has limitations and there is the risk of tool misuse. This paper suggest suitable applications for 
auditing, outlines training options for different classes of auditing and discusses risks associated with abuse 
of the audit tool. A general audit model is illustrated in Figure 1. An audit process used by NASA at the 
Kennedy Space Center is at Attachment 1 to this paper. Specific instruction for an audit process is out of 
scope. 


Auditing Practice 

The auditor has three customers: (1) The company, (2) The auditee, and (3) The audit program manager. 

A balancing effort is common place because only rarely are the needs of the three customers congruent. 

Two classes of audits: (1) The traditional audit compares known requirements against evidence that 
requirements are met; and (2) Internal audits critically evaluate a business process searching for process 
improvements. Traditional audits are specific and are normally performed by personnel with auditing in 
their job description, while internal auditors are often subject matter experts who may or may not have had 
audit training and are on a team led or facilitated by a professional auditor. The internal audit is less 
specific than the traditional audit. The audit definition above, originating from the quality program, falls 
short of meeting the intent or scope of the internal audit. 

Audits typically take one of two forms: (a) Scheduled audits which are somewhat routine, and as the name 
implies, are audits that are scheduled throughout a specific time period, (b) The second type of audit is an 
evaluation that responds to a real-time problem - similar as a response to a mishap. 

Historically, safety professionals have used both forms of audits. The question is - can safety audits 
produce more meaningful information? Or more consistent information assuming consistency is 
important? And more trendable data? 

Safety Programs claim to use the audit, but when I supervised an office of both quality specialists and 
safety specialists, both conducting audits and producing audit reports, only then did I realize that the quality 
specialists out-preformed the safety specialists in the audit arena. Fundamental talent assumed equal 



between the two fields, training and experience accounted for part of the differing results between the two. 
The better thought out quality logic contributed as well. 
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Figure 1- Typical Audit Model 


Review of the Quality Program implementation of audits provides useful insights. First, the Quality 
Program views the audit process at different levels. The Quality Specialist conducts audits that determine 
if requirements are met - no more and no less. Typically these first level audits are performed against a 
checklist composed of all requirements of interest. These checklists insure audit scope is maintained and 
consistency among different auditors assured. When the level one audit by the quality specialist is 
complete, a second audit can be scheduled to validate a process. This level two audit is also performed 
against a checklist. Quality Engineering, who manages requirements, have the option to conduct audits to 
determine value added of existing requirements or if new processes reflect new requirements which 
represents a middle ground between the classic traditional audit and the internal audit that is practiced by 
many of today's corporations. 


Metrics 


Metrics are useful if progress is to be measured. One process to evaluate the contractor safety profile at 
NASA's Kennedy Space Center (KSC) is presented at the 2nd attachment. One good attribute of the KSC 
process is that it incorporates a binary metric which saves time by reducing unproductive discussion or 








debate related to the subjective metric that the contractor is assigned for a given rating period. An 
interesting logic of this process that has been used at KSC is that each attribute has a favored position in the 
center of a balancing act where at each side of center is an undesired position that is equal and opposite, 
resembling a physics vector. 


Training and Examples 

NASA’s Space Shuttle contractor, the United Space Alliance (USA), at the Kennedy Space Center (KSC) 
recently, on its own internal initiative, developed a process for self-evaluation or self-improvement. The 
contractor attacked this challenge with vigor - pooling some of its best employees who could provide useful 
consulting. Finally, the contractor provided both training and teambuilding to prepare this team for its 
challenge. The contractor completed several “internal audits” and the interim results were excellent. One 
novel idea woven into their process is to define at least one role player on an audit team: the role of the 
“devil’s advocate”. This role is to question everything and because its defined as a role, debates are de- 
personalized and become more constructive. This Devil’s advocate role also serves to reduce the 
possibilities that the audit group will develop undesired tendency for groupthink as reported in research by 
Irving Janis in his book, Groupthink . (Ref. 3) “Groupthink refers to a deterioration of mental efficiency, 
reality testing, and a moral judgment that results from in-group pressures.” Janis list 7 decision making 
defects resulting from groupthink. For example, Janis states, “Seventh, the members spend little time 
deliberating about how the chosen policy might be hindered by bureaucratic inertia, sabotaged by political 
opponents, or temporarily derailed by the common accident that happen to the best of well-laid plans. 
Consequently, they fail to work out contingency plans to cope with foreseeable setbacks that could 
endanger the overall success of the chosen course of action.” - a non-obvious risk easily overlooked by the 
audit team. 

One government organization with experience and recognition for doing internal audits well is Sandia. 
Southwest Airlines is one commercial entity with a strong internal audit group according to Institute of 
Internal Audits. 

Audit Training Resources : A training class recommended by the American Society for Quality (ASQ) 
titled “Quality Audits for Improved Performance” (Ref. 4) was very good. It was taught by Dennis R. 

Arter. KSC scheduled this class for Shuttle safety personnel and process analysts. The audit company is 
Columbia Audit. 

NASA Headquarters contracted with the Institute of Internal Auditors to present an “Operational Audit” 
class (or internal audit training) to KSC as test training. All NASA Centers were invited to evaluate the 
training. If such training is useful to your organization, the point of contact in the Institute of Internal 
Auditors’ company is Denise Johnson (407-937-1337). 

The ISO 9000 Lead Investigator training, in my opinion, provided little broad based skill or logic that 
safety personnel could apply to conducting its audits. 

At the conclusion of these two different types of audit training (traditional auditing vs. internal auditing), 
civil servant personnel and contractor personnel who attended both classes compared the two. A summary 
of our brainstorming session, Figure 2, identifies desired attributes of the audit training and compares the 
two recent and different training classes. From this government-contractor team’s point of view, if one 
attribute is favored, it is marked by a “+”. 

Correlation Between Mishap Investigation Logic and Audit Logic 

At the International System Safety Conference 2003, a relation of logic between internal audits (operational 
audits) & mishap investigations became apparent. During one of the presentations on mishap investigation, 
subsequent questions and dialogue identified a Canadian company who (a few years past) asked if it were 
feasible if mishap in vestigation training/techniques could be used for conducting internal audits. This 
company planned an internal audit process founded on mishap investigation training. The kinship between 
audit training and mishap investigation training became apparent. 




Mishap investigations typically contain well thought out analyses and conclusions. However, more often 
than not, the recommendations have less depth and justification, and seem just stuck on as an afterthought. 
By comparison, internal audit teams' recommendations are table topped by both the audit team and a group 
of independent advocates to insure good recommendations are listed. If good recommendations were not 
generated, then no recommendations are included. Recommendations obvious to everybody only serve to 
dilute innovative recommendations - the crown jewels of the internal audit effort. More likely, the better 
recommendations come from the organization being audited who are addressing/responding to the concerns 
of the auditors. 


Institute of Internal Auditors 

Attribute 

Columbia Audit (D. Arter) 
[Traditional audit logic] 

Likely a product of several 
experienced individuals which is 
good 

Training Class Organization 

+ Highly structured, but likely a 
product of one individual. 

High level - Deciding what to audit. 
Need instruction on how to audit. 

Audit Process 

Complimentary — 

Nuts & bolts - how to audit 

+ Risk Based 

Decision 

Not applicable 

Planning projected as 50-70% of 
audit and relates to defining scope 
and emphasizing communication 
with management 

Communication 

Planning projected as 25% of audit 
& relates to research of 
requirements and flowcharting 

« 

rr' Complimentary — — 

How to write findings - good 

Soft controls emphasized 

Control (formal vs informal) 

Soft controls addressed but not 
emphasized 

Unstructured. Dependent on 
experience of instructor. Examples 
were compliance type from finance 
world. Real engineering examples 
needed. 

Examples 

+ Structured & realistic with role 
playing. 

Internal audits should have 
recommendations 

Recommendations 

Rare 

Too much time at beginning of class 

Notes 

Excellent very focused instructor 

with introductions. 


with a highly organized case study 
used to tie the course concepts 
together 


Figure 2 - Audit Approach Comparison 


Audit Program Risks and Shortcomings 


The traditional audit is based on the false premise that the founding fathers of the system or process were 
all knowing and generated perfect timeless requirements. Audits performed against requirements are self 
limiting and therefore can not lead to system or process improvement or leads to process improvements 
only with great difficulty. These traditional or compliance audits do not challenge the continued validity of 
the requirements - representing a rut or paradigms where many organizations fall into obsolesce. An 
auditor frequently using words such as nonconformance, discrepancy, problem resolution implies a 
traditional or compliance audit it being performed and not an internal audit. 

Audit program risks are reduced when the audit is conducted avoiding various traps: 











© Emotional words and phrases 

• Bias and slanted viewpoints 

© Insignificant issues 

© More than six findings 

• Poor interview technique 

Management Concepts Incorporated provides training in contracting. The class handout page 4-17 (ref. 1) 
offered this advice, “Further, agencies must, to the maximum extent practicable, avoid relying on 
cumbersome and intrusive process-oriented inspection and oversight programs to assess contractor 
performance.” Management Concepts Incorporated addressed unreasonable interference with contractor’s 
work on page 13-4.' “Although the government has broad rights in the inspection process, it has been held 
liable if it exercises these rights in a manner that unreasonably interferes with the performance of the work 
by the contractor or that increases the amount of work required of the contractor. There have been a 
number of cases where the government has been held liable for unreasonable interference, such as: 

• Inconsistent, multiple inspections [or audits]; 

© Extremely rigid, unreasonable, and arbitrary conduct of the Contract Officer Representative 
(COR) [a COR could be a safety representative]; 

• Overzealous supervision of work by the COR; 

• Confusing and vacillating inspection [or audits] procedures; 

• Multiple inspections [or audits] to differing standards by different CORs; and 

• Overly close surveillance, inordinate number of visits by CORs, and failure to cooperate in 
providing inspection when needed. 

In these cases the contractor claimed breach of the implied duty of cooperation and was awarded damages 
by the court.” 

The process for handling the Independent Assessment Report is critical to the success of the program. In 
“The Essential Drucker”, (ref. 2) pages 120-122 from the section on Self-control through Measurements, 
Peter Drucker states, “That information can be effectively used for self-control is shown by the example of 
General Electric. General Electric has a special control service - the traveling auditors. The auditors study 
every one of the managerial units of the company thoroughly at least once a year. But their report goes to 
the manager of the unit studied. There can be little doubt that the feeling of confidence and trust in the . 
company that even casual contact with General Electric managers reveals is directly traceable to this 
practice of using information for self-control rather than for control from above. But the General Electric 
practice is by no means common or generally understood. Typical management thinking is much closer to 
the practice exemplified by a large chemical company. In this company a control section audits every one 
of the managerial units of the company. The results of the audits do not go, however, to the managers 
audited. They go only to the president, who then calls in the managers to confront them with the audit of 
their operations. What this has done to morale is shown in the nickname the company’s managers have 
given the control section: ‘the president’s Gestapo.’” 

Summary 

Audits can be a useful safety tool if complemented with appropriate training, thoroughly planned, with a 
disciplined scope, and executed with sound judgment of mind. But then the same can be said of all safety 
tools and processes. Good luck with your audit. 



CONTRACTOR SAFETY PROFILE 


OBJECTIVE: Document Contractor safety profile. 

Two scales are used to estimate or define Contractor safety profile: (1) Contractor Processes (2) Contractor 
employee behavior. Each of these scales is composed of several attributes. 


Contractor Process or Change Scale 


Stagnation 


Acceptable Ideal Acceptable 

£ == &--- = z£ 


Chaos 


Apathy 


-1 0 +1 0 
Contractor Employee Behavioral Scale 
Acceptable fdeal Acceptable High Risk Behavior 


-1 


-1 

(or poor planning) 


0 


+1 


0 -1 

(or not understood risk) 


REQ. 

ATTRIBUTES 

CONTRACTOR 

ACTIVITY 

Grade 


Management & Employment Commitment 
(Element 1) 

Reporting of safety concerns implies ideal 
behavior. 

Healthy volume of reporting ideal. Non reporting 
implies apathy. 




Management & Employment Commitment 
(Element 1) Communication. Active discussion of 
safety risks implies the ideal. 




Management & Employment Commitment 
(Element 1) 

Employee discipline is ideal. 

Respect for hazardous operations control areas is 
ideal where disregard for control areas is high risk 
behavior. 




Management & Employment Commitment 
(Element 1) 

Requirements management. 

Excessive number of safety variances imply poor 
planning. However, safety variances resulting 
from conscious choices where documentation 
resulted in no change in risks are acceptable; and 
safety variances to rigid rules where innovation 
developed a safety method with less risk than 
residual risk associated with the rule is 
commendable. 




Worksite System & Analysis (Element 2) & 
Hazard Prevention & Control (Element 3) 











Hazard Prevention & Control (Element 3 ) 
Contractor mishap rate as compared to like 
industry mishap rate. Contractor forms mishap 
investigation boards more often than when 
required by NASA. 


Hazard Prevention & Control (Element 3) 
Violation of safety requirements. 

Violation of safety requirements is either poor 
planning or high risk behavior. Example of an 
operations violation is disregard for safety control 
during hazardous operations. 


Safety & Health Training (Element 4). 

Adequate employee qualifications insures 
recognition and understanding of safety risks. 

See safety employee qualifications at time of 
hiring, and review all employee training records. 
Certified Safety Professional (CSP) registration or 
PE registration, annual safety training courses and 
VPP certification are examples of qualified and or 
a safety motivated work force. 


Potential personnel actions. 

Company growth is ideal while potential layoffs 
results in high risk behavior or chaos. 


Contractor process change. Shuttle is a mature 
and stable system with little uncertainty. Expect 
no revolutionary change, only expect methodical 
Demming type changes. 

Healthy processes should be constantly 
reevaluated, (a) Quality inspections where there 
are no changes imply stagnation and huge changes 
to quality inspections imply chaos, (b) Active 
Contractor Corrective Actions imply the “ideaT 
(c) Contractor innovation affecting safety is ideal. 



Attachment 1 





AUDIT PROCESS 


Objectives 

- Provide a method to efficiently perform contractor 
audits for compliance to documented requirements 


Requestor - Method to verify contractor's capability to conadently 
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